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Abstract A structured search of library databases revealed that research examining the effectiveness of e-Learning has 
heavily increased within the last five years. After taking a closer look at the search results, the authors discovered that 
previous researchers defined and investigated effectiveness in multiple ways. At the same time, learning and development 
professionals within public and private organisations are increasingly being asked to prove the effectiveness of their 
learning and development initiatives. This paper investigates the effectiveness of e-Learning through an integrative review. 
The paper answers the following research questions: How is the effectiveness of e-Learning defined? How is the 
effectiveness of e-Learning measured? What makes e-Learning solutions effective? The authors discovered 19 distinct ways 
to define effectiveness, the most common of which is 'learning outcome', appearing in 41 % of the articles examined in the 
literature review. Moreover, the most common way to measure effectiveness is quantitatively with pre- and post-tests. 
This paper includes an empirical study of an e-Learning solution for science teachers (K—12) which serves as a valuable 
addition to the findings of the literature study. The study suggests that it is difficult to use e-Learning to improve teaching 
performance, as participating teachers can apply several strategies to avoid substantially changing their work-related 
practices. Furthermore, the study shows that only using the fulfilment of pre-defined learning objectives as an 
effectiveness parameter does not allow developers and researchers to see unexpected and unintended changes in practice 
that occur as a result of the e-Learning program. Finally, the research provides insight into the validity of self-assessments, 
suggesting that participants are able to successfully report their own practices, provided certain qualitative survey 
approaches are used. In this paper, a model for understanding the relationships of the key factors that influence 
effectiveness is developed. The model categorises these factors from three perspectives: the context in which the e- 
Learning solution is used, the artefact (the e-Learning solution itself) and the individuals that use the artefact. It was found 
that support and resources, the individuals' motivation and prior experience and interaction between the artefact and the 
individuals that use it all influence effectiveness. Finally, this paper discusses whether e-Learning and traditional face-to- 
face learning should be measured according to the same definitions of and approaches to effectiveness, ending with a call 
for learning designers and researchers to target their measurement efforts to counting what counts for them and their 
stakeholders. 
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1. Introduction 

Research examining the effectiveness of e-Learning has increased in recent years. This is primarily due to the 
increased possibilities for IT and learning as well as increased political and organisational attention to 'what 
works' in learning. Figure la shows the 761 papers relevant to this research, and Figure lb shows 111 
intensively coded abstracts of the 761 papers (which are described in further detail in the methodology section 
below). There are fewer papers published in 2013 than in any other year because the structured search took 
place in October 2013. 

In the following analysis, the authors investigate the research into the effectiveness of e-Learning. The paper is 
structured around three research questions: How is the effectiveness of e-Learning defined? How is the 
effectiveness of e-Learning measured? What makes e-Learning solutions effective? The aim of the literature 
study is to organise similar research in order to better understand the characteristics and tendencies as well as 
connections between the applied concepts. 
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Figure la: Number of published papers per year 
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Figure lb: Number of published papers with coded abstracts per year 

1.1 Literature study - methodology 

Several systematic reviews and meta-studies on the effectiveness of e-Learning are considered within the 
context of health care or language learning. These reviews primarily include quantitative studies based on 
certain criteria, such as sample size (Veneri, 2011), transparency of statistical information (Grgurovic, Chapelle 
and Shelley, 2013; Means et a I, 2013) or homogeneity of the respondents and predefined outcome measures 
(Rosenberg, Grad and Matear, 2003). Only one relevant meta-review, which included both qualitative and 
quantitative studies in an integrative review evaluating the outcome of distance learning for nursing 
education, was found (Patterson, Krouse and Roy, 2012). 

The quantitative meta-reviews aimed to document the effectiveness of e-Learning by consolidating the data of 
a number of quantitative studies. The mixed-method meta-review mentioned above describes the state of the 
research, explains how the studies evaluate different outcomes and discusses different aspects of learning 
effectiveness. This is somewhat similar to the present paper, which also applies a mixed-method methodology 
in an integrative manner. However, many more research articles are considered in this paper due to broader 
selection criteria. Hence, this paper is not concerned with re-investigating how effective e-Learning is, but 
rather with understanding the definitions, measurements and factors promoting e-Learning effectiveness. 

The authors aimed to obtain a broad foundation of high-quality papers, from which a large but not pre-defined 
number was chosen for further investigation. Papers were chosen using a strategic randomised approach 
based on a purposive sample size, then analysed based on the concept of theoretical saturation (that is, the 
point at which new data no longer provide further insight into the subject at hand). In this integrative review, 
data analysis, data reduction and data displays are equally important (Whittermore and Knafl, 2005). 

The authors conducted conventional subject searches in 30 academic databases (J-stor, Scopus and Proquest, 
which includes 28 databases) to discover articles examining the effectiveness of e-Learning within the context 
of adult learning (see Table 1). All fields of research were included in the searches, as e-Learning can be used 
to support any subject. The searches only included articles in English, and where possible, only peer-reviewed 
journals. The chosen synonyms for 'effectiveness' include 'transfer' and 'application', which may have resulted 
in an overrepresentation of articles that define effectiveness as the application of learning content into work 
practices. 

The searches resulted in almost 1000 articles. Articles clearly irrelevant to the subject were excluded, 
diminishing the number to 761. If an article contained an empirical study on the effectiveness of e-Learning 
and the solution under investigation was targeted at working professionals or students, then the abstracts 
were carefully coded and analysed in great detail using Nvivo 10. When doubts about the relevance or coding 
of the abstracts surfaced, the two authors discussed the abstract, decided on the best coding and documented 
what was learned from the discussion in a shared document. 
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Table 1: Applied search string 


effect* OR transfer* OR applica* OR impact OR outcome*(search title) 

AND 

"e-Learning" OR e-Learning OR online OR web-based OR "web based" OR technology* OR WBT OR WBL OR blended OR 
Internet OR Distance OR CBT OR CBL OR distance OR Computer OR mobile OR simulation* OR "social media" OR 
"community of practice" OR game* OR gamification* (search title) 

AND 

learning OR training OR education OR development OR "competence development" (search title) 

AND 

adult OR "competence development" OR lifelong OR profession* OR employee* OR worker* OR "further education" 
OR master OR business* (search abstract) 

NOT Children OR Child OR kids OR Youth OR "Technology transfer"(search anywhere) 


Before the coding began, a rough coding scheme was created based on the research questions, which entailed 
parent codes (named 'nodes' in NVivo) such as 'definition of effectiveness', 'research question', 'research 
methodology', 'subject area', 'audience', 'theories applied', 'technology applied',' key findings' and so on. The 
detailed coding tree was created through in vivo coding, a grounded approach in which codes are added as the 
analysis reveals relevant factors by using the original statement of the source as a code name (Harry, Sturges 
and Klingner, 2005). New sub-nodes were continuously created as new definitions, new factors of 
effectiveness and new technologies were discovered in the abstracts. 

As mentioned, the purposive sampling led to a strategic randomised approach for selecting papers from the 
large collection. The papers were not investigated in alphabetical order. Rather than analysing from A to Z, a 
variety of letters in the alphabet was chosen, as there tend to be surnames, and henceforth letters, which are 
used more in some regions than others. Though the aim was not to obtain an even global distribution of 
papers, the authors, newertheless, tried to accumulate the broadest scope of information possible. 

Of the 224 abstracts that were carefully read, 111 fulfilled all the criteria and were coded in detail using the 
above-mentioned method. 

1.2 Empirical study - bringing context into the literature study 

The empirical study aimed to discover if, how and why an e-Learning program would be successful in 
improving science teachers' work practice in Danish elementary schools. Thus, the empirical study lives up to 
the criteria of the literature study, as it focuses on the effectiveness of e-Learning for working professionals. It 
also explores some of the challenges highlighted by the literature study. 

The solution and learning design focused on developing competent teaching methods for natural science. In 
this project, effectiveness is understood as the transfer of learning, which positively impacts teaching 
practices. The e-Learning was investigated thoroughly from February to June 2014 as a possible solution for 7 
teachers at three Danish elementary schools. 

The data gathering method included extensive in-class video recordings and observations. The researcher 
recorded teaching methods using a mobile ethnographic approach; the teachers had small camcorders 
attached to their necks, which enabled the researcher to view the classroom from the teachers' perspective. 
The data consisted of approximately 120 hours of in-class video recordings and 100 pages of observation 
notes. The researcher also had reflection sessions with the teachers before and after the introduction of e- 
Learning. These sessions were inspired by the mind tape and retrospective interview methodologies (as in 
Kumar, Yammiyavar and Nielsen, 2007). The teachers' interactions with e-Learning, including their preparation 
for classes, were recorded by Camtasia, a software program that can record the screen, mouse movements 
and a picture-in-picture setting of the user. Here, the think-aloud approach was applied (Nielsen, Clemmensen 
and Yssing, 2002). The data consisted of approximately 25 hours of video recordings and 40 pages of 
observation notes of teachers' interactions with e-Learning. The teachers responded to a satisfaction survey 
immediately following the conclusion of the e-Learning, as well as a pre-survey shortly before initiating the e- 
Learning and a post-survey approximately one month after completion of the e-Learning. The latter was 
repeated 6 months after completion. This final data consists of 28 responses to the surveys, which each had 
approximately 20 questions. 
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The approach to gathering empirical data was specifically designed to capture some of the complexity, 
possibilities and challenges of teaching practices, both expected and unexpected. In the following section, the 
preliminary results of the empirical study are included when they contribute to answering the research 
questions of this paper. 

2. How is the effectiveness of e-Learning defined? 

Approximately one-third of the abstracts in the literature study were coded. From these, as many as 19 
different ways to define effectiveness have been identified. These definitions are listed in chronological order 
below, in Table 2, with the most commonly used definition at the top. The table includes 92 papers of the 111 
coded. The remaining abstracts did not state the target audience and are therefore not included in this table. 
The actual number of papers from which the 19 definitions were obtained is 170, not 92. This is because a set 
of definitions is often used to investigate the effectiveness of an e-Learning solution; for example, several 
papers use both 'learning outcome' and 'satisfaction' as definitions for effectiveness (Harrington and Walker, 
2009; Jung et al, 2002; Maloney et a I, 2011). The number of papers in this list would of course change if the 
remaining abstracts were coded, but the author's find that the most common definitions are expected to stay 
relatively stable, as they have not significantly changed in recently reviewed abstracts. 

Table 2: Definitions of effectiveness organised by the context of adult learning 



Higher 

education 

Work-related 

learning 

Total 

Number of papers: 

52 

40 

92 

Distribution of papers: 




Learning outcome 

29 

9 

38 

Transfer (application to practice) 

3 

15 

18 

Perceived learning, skills or 
competency 

11 

6 

17 

Attitude 

8 

3 

11 

Satisfaction 

8 

3 

11 

Skills acquired 

5 

5 

10 

Usage of product 

4 

5 

9 

Learning retention 

4 

4 

9 

Completion 

0 

5 

5 

Motivation and engagement 

3 

2 

5 

Organisational results 

0 

5 

5 

Application to simulated work 
practice 

0 

4 

4 

Self-efficacy 

0 

4 

4 

Confidence 

1 

2 

3 

Cost-effectiveness 

1 

2 

3 

Connectedness 

1 

1 

2 

Few errors 

2 

0 

2 

Raised Awareness 

0 

2 

2 

Success of (former) participants 

1 

0 

1 

Undefined effectiveness 

10 

2 

12 


Of the papers reviewed, 57 % (52/92) examined effectiveness within higher education, in which context the 
most prominent definition of e-Learning effectiveness was 'learning outcome', with 56 % (29/52) of these 
papers applying this definition. Within work-related learning, the most common definition was 'transfer 
(application to practice)', with 38 % (15/40) of the papers applying this definition. 

'Learning outcome' occurs when participants acquire new understandings as a result of the e-Learning 
initiative. This is a broad definition, but in the abstracts of papers examining higher education, the definition is 
often clarified in terms of measurements; for example: 'Student learning measurements included: pre-test, 
final examination (post-test) and final letter grade' (Boghikian-Whitby and Mortagy, 2008). 

Within the field of work-related learning, the ability to apply the content or processes of the e-Learning 
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solution is essential. For example, in a study on teachers' technology competencies, it was not 'knowing 
about', but rather the actual 'integration of computer activities with appropriate inquiry-based pedagogy in 
the science classroom' that determined effectiveness (Angeli, 2005). 

It is, however, interesting that 'transfer (application to practice)' is sometimes evaluated through the 
participants' self-assessments: 'Outcomes were measured across levels 1 to 3 of Kirkpatrick's hierarchy of 
educational outcomes, including attendance, adherence, satisfaction, knowledge and self-reported change in 
practice' (Maloney et al, 2011) and 'A follow-up questionnaire showed that two-thirds of those who viewed 
the program had subsequently reviewed the performance data for the initial wire they were using and 20 % 
had changed it' (Marsh et al, 2001). This brings to light the discussion of whether or not it is possible for 
learners to assess their own transfer (i.e. if people accurately report their actions, or if researchers, managers, 
peers or learning professionals must observe and report). 

Since learning literature often focuses on engagement and motivation as necessary factors for knowledge gain 
and learning transfer, it is surprising that only five papers include these aspects in their research (Table 2). 

Some papers investigated the interrelatedness of more aspects of effectiveness, such as the relation between 
learning outcome/retention and behaviour. For example, Hagen et al (2011) found that '...the effects of the 
intervention on security awareness and behaviour partly remains more than half a year after the intervention, 
but that the detailed knowledge on information security issues diminished during the period.' Such a study 
challenges the idea that behaviour changes can be measured through learning retention. 

Table 2 also shows that the abstracts dealing with higher education operated with few definitions other than 
'learning outcome', while the abstracts dealing with work-related learning generally applied a greater variety 
of definitions. This could be because universities work with performance requirements that primarily focus on 
examination grades and completion rates, causing effectiveness to be measured by cognitive knowledge 
indicators. In a work-related setting, however, effective learning is much broader, including aspects that are 
not bound to individual or group projects, such as the application of learning to work contexts, organisational 
results and cost-effectiveness. 

It became clear in the analyses that many abstracts and a number of papers did not state their definitions of 
effectiveness; 13 % (12/92) of the abstracts left effectiveness completely undefined. 

2.1 Why is this important? 

Having multiple ways of understanding the effectiveness of e-Learning allows professionals and researchers 
substantial flexibility in defining, measuring and determining the effectiveness of an e-Learning solution. 
However, the broadness of the concept does present challenges. Leaving the concept of e-Learning undefined 
may lead to misunderstandings, and the aspects of effectiveness that are of most value to participants and 
stakeholders may not be considered. Illuminating the many definitions of effectiveness can lead to reflection 
and inspiration for appropriately utilising the concept of effectiveness, thus enabling learning professionals to 
better align their expectations and target their measuring efforts towards what is important to them and their 
stakeholders. 

3. How is the effectiveness of e-Learning measured? 

The previous section broadens the understanding of the definitions applied within research examining the 
effectiveness of e-Learning. But how are these definitions investigated in the various studies? How do the 
researchers measure effectiveness, and what consequences result? Of the 111 abstracts coded in detail, 63 
abstracts identified their research design. 

Table 3: Research study methods 



Mixed 

Qualitative 

Quantitative 

All abstracts coded with... 

9 

5 

37 

Comparative studies applying... 

0 

1 

18 


The first row of Table 3 shows the distribution of research studies coded as mixed, qualitative and quantitative 
studies. In addition, 30 comparative studies were found, 11 of which do not describe in the abstract whether 
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they are conducting qualitative, quantitative or mixed methods research. The distribution of the rest is shown 
in the second row. Nearly 73 % (37/51) of the studies are quantitative. Almost half of these are comparative 
studies, which compare e-Learning with traditional face-to-face and/or blended learning. The vast amount of 
comparative quantitative studies may be due to policy makers' interest in this research (Grgurovic et al, 2013). 

The literature study reveals that the most common way to measure effectiveness is through quantitative pre- 
and post-testing. To come to an understanding of which definitions of effectiveness are most used in particular 
kinds of studies, the effectiveness code was correlated with the research methods applied. This correlation 
showed that 'learning outcome' was used more frequently in quantitative studies (18 papers) than qualitative 
(2 papers) and mixed methods studies (1 paper). More quantitative studies were identified than qualitative 
studies, but the quantitative studies' use of 'learning outcome' is still significantly higher. 

It might be assumed that qualitative studies would use several definitions of effectiveness, but this was not the 
case. Instead, these studies tended to use only one or two of the 19 definitions, whereas quantitative studies 
used significantly more. This could be because qualitative and mixed methods studies aim to explore a single 
concept in dept, and the intentions are often to understand the 'why's' of such a concept, which requires a 
significant amount of time and resources. On the other hand, quantitative research uses definitions as a set of 
variables constituting effectiveness, thus necessitating the use of several definitions. 

The reason for the distribution of research methods in this literature study could be due to both a publication 
and policy bias. Writing thorough descriptions of the 'why's of qualitative research requires more space than 
reporting means and standard deviations. Very few journals allow for such prolonged papers, and quantitative 
papers also tend to be in higher demand, in line with what Grgurovic (2013) calls a 'publication bias' (i.e. the 
tendency to only publish studies with statistically significant findings). 

3.1 Why is this important? 

As stated, most research into the effectiveness of e-Learning focuses on measuring if and/or which e-Learning 
solutions are effective using quantitative measures. The aim of the empirical study examining an e-Learning 
program for science teachers was to understand the complex approaches used, when attempting to change 
teaching practices using e-Learning. The solution uses an on-the-job learning approach, including in-class 
practice, and a facilitated team-based competence development setup. It was shown that great effort is 
needed to use e-Learning to improve teaching performance. 

The qualitative analysis of the teachers' interactions with e-Learning (Camtasia recordings) shows three 
prevailing strategies that the teachers use to avoid substantial changes to their work practice: 

1. Finding statements to reject content, which means that the teachers seemed to be searching for single 
statements in the e-Learning content that they could use to prove that application to practice was not 
possible. Some stated that they preferred to teach as the e-Learning suggested, but their work context 
would not allow for it. 

2. Modifying content to make change less demanding, which means that the teachers consciously or 
unconsciously modified the content to work similarly to their current practices, allowing them to state 
that they were already teaching this way, or changing the content to become easily applicable. This 
finding is in line with Bransford and Schwartz (1999), who discovered that people often modify a transfer 
situation until it becomes similar to something they know (Lobato, 2006). 

3. Pinpointing content that can be easily implemented, which means that the teachers used elements of the 
content that they could easily apply to their teaching without changing it fundamentally. 

For the quantitative and qualitative surveys, the teachers were asked to evaluate their application of the 
program's learning content to the lessons that were observed by the project researcher. This enabled the 
researcher to compare the teachers' self-assessments of transfer and transfer-related concepts (motivation, 
knowledge and self-efficacy) with the observation material. This led to the conclusion that all teachers 
following the program made noticeable changes to their teaching practice, largely using the third strategy 
mentioned above. 
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This research design (see 1.3) also enabled the researcher to discover unintended and unexpected transfer. 
For example, one teacher became so fond of her new way of posing questions to the pupils that she now uses 
the method when teaching history as well. On the other hand, her co-worker was insecure with the new 
teaching methods, which negatively affected her teaching. Research examining learning transfer shows that 
the traditional notion of one-to-one transfer from learning to practice must be challenged (see Lobato, 2006). 
A challenge often faced when evaluating effectiveness is that unexpected transfer, which can have both 
positive and negative impacts on performance, may not be analysed if only known and a priori concepts are 
investigated. Thus, if only quantitative survey data was gathered, the empirical study would have presented a 
misleading view of the transfer of learning. In addition, the teachers generally overestimated themselves in 
both pre- and post-testing. However, by including the qualitative elements of the survey (e.g. teachers 
describing their actions during the lessons in their own words), most discrepancies between self-assessment 
and observation were clarified, and the responses could be accepted or corrected. The amount of pure 
quantitative research in the literature study was also of concern. Results relying solely upon rating scales and 
multiple-choice tests can easily become misleading. Openness to participants' own unframed understandings, 
even if only part of a survey, can potentially result in more valid and usable answers regarding the 
effectiveness of e-Learning, regardless of its definition. 

4. What makes e-Learning solutions effective? 

All the abstracts used in this study were coded for whether the e-Learning was effective, not effective or partly 
effective, provided this was stated or indicated in the abstracts. This was the case for 61 of the 111 abstracts 
examined. The distribution is shown in Table 4. 

Table 4: Is e-Learning effective? 


Effective 

41 

Not effective 

6 

Partly effective 

14 


Considering the challenges of e-Learning, the fact that only 10 % (6/61) of the studies are classified as 'not 
effective' questions the validity of the classifications. Taking a closer look at the abstracts, it became clear that 
many of the empirical studies on effectiveness were conducted by researchers who appeared to have a stake in 
the success of e-Learning. This issue of 'effectiveness bias' means that the literature study at this point does not 
support the investigation into which e-Learning solutions are particularly effective. Perhaps a future analysis of 
the papers in question is warranted. What this study can explore are the factors that influence e-learning 
effectiveness. 

A qualitative view of the factors, which the researchers classify as either promoting or prohibiting e-Learning 
effectiveness across a spectrum of definitions, methodologies and e-Learning media, provides valuable 
additions to e-Learning design and research. Through in vivo coding, 34 factors were found and divided into the 
three categories: individual (subject), contextual scaffolding (context + object) and e-Learning solution and 
process (artefact). These categories are inspired by the concept of activity theory, as they relate to learning 
and the transfer of learning (in line with Engestrom, Leont'ev, Vygotsky and Orlikowsky). 


Table 5: Factors that influence effectiveness 
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E-Learning solution and process 

Individual 

Context scaffolding 

(artifact): 

(subject): 

(context + object): 

Active learning 

Age 

Learning environment 

Applicable to practice 

Entrance scores 

Support 

Balance between asynchronous and 

Experience in profession 

Technological resources 

synchronous activities 

Language (second language) 

available to user 

Cognitive load 

Learner characteristics 

Time available to learn 

Collaboration 

Learner preferences 


Communication 

Locus of control 


Computer playfulness 

Motivation 


Design 

Expected workload 


Instructional scaffolding 

Perception of learning 


Interaction (learner - instructor and 

Previous e-Learning experience 


peer2peer) 

Previous training 


Learner control 

Prior online experience 


Modeling 



Problem-based learning 



Practice (incl. case study, case-based 



learning, and simulation) 



Structure 



Technological constrains 



Usability 




The categorization of the factors shows an interesting distribution (Table 5). The papers examined in the 
literature study clearly prioritise factors related to the e-Learning solution and process, even though 
contextual factors may be more critical to e-Learning effectiveness (Noesgaard, 2014). The reason for this 
phenomenon may be that the contextual factors are perceived as too complex and changeable to investigate 
and control for research, and that they lay outside the responsibility of learning professionals. 

As previously mentioned, the grounded approach uses the wording from the papers in the first round of 
analysis. This was also done for the factors listed in Table 5. In the second round of analysis, the authors found 
that the interconnectedness of the factors called for further categorisation. This categorisation into key factors 
is discussed further below and is illustrated in Figure 2. 

Overall, in terms of the contextual factors, the key factors are quite clearly 'resources' (time, technology) and 
'support' (from managers, IT personnel or peers) in the learning environment. These factors are essential for 
using e-Learning initiatives to improve performance and change behaviours. 

With regards to individual factors, the papers generally agree that effectiveness varies according to individual 
differences (e.g. Armatas et a I, 2003; Aydogdu Karaaslan, 2013). Some papers refer to learner characteristics 
broadly and others discuss particular issues relevant to their study. Two mentioned characteristics are 'age' 
and 'previous online experience' (Table 5). One study suggests '...that adult students benefit more from taking 
online classes compared to traditional age students [...] and that computer competency helped improve 
performance in online classes over time' (Boghikian-Whitby and Mortagy, 2008: 107). Sometimes, factors that 
are not mentioned can have an impact on effectiveness: 'However, although gender is a significant predictor in 
traditional classroom courses, its effect disappears in Web-based courses. There is evidence that Web-based 
courses can be conducive to the leaning process of technical knowledge for female students' (Lam, 2009). 

The individual factors largely fit into two categories related to learner characteristics: 'experience' and 
'motivation'. 

It is not surprising that the experience of participants, in terms of previous relevant work experience and 
online experiences, affects the effectiveness of e-Learning. These factors seemed to determine the kind of 
attitude that participants 'go into the learning process with'; previous experience can be beneficial, if the 
previous work and online experiences correspond with the e-Learning (Boghikian-Whitby and Mortagy, 2008; 
Bennison and Goos, 2010; Haverila, 2010). What is intriguing is that, experience may either increase or 
decrease effectiveness. The authors have in previous empirical studies within higher education with students 
who study IT and educational design found that, when the definition of effectiveness was 'satisfaction', the 
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students' previous experience with a variety of e-Learning solutions left the students unimpressed; hence, 
decreasing effectiveness (satisfaction). In the empirical study discussed in this paper, the teachers had little e- 
Learning experience and were generally satisfied. Similarly, a learner with significant relevant work experience 
may perform well on tests and at work but have a low satisfaction score because the learning was not 
challenging. This underlines the importance of clarifying the definition of effectiveness and to discus the ways 
in which individual factors can affect effectiveness. 

Motivation to learn and engage with the e-Learning solution is key to effectiveness, especially when 
effectiveness is defined as the time spent using the product: 'Results suggest the importance of motivation to 
learn and workload in determining aggregate time spent in e-learning courses' (Brown, 2005: 465). However, 
when projects are defined as 'blended e-Learning', time spent may not always be a good indicator of whether 
learning took place: 'beyond the impact of extrinsic-related perceptions, social and personal motivations are 
important drivers of discussion forum usage in an e-learning context...[] ... It is concluded that even for adult 
learners, social interaction with instructors and collaborative interaction with peer students are important in 
enhancing learning and active participation in online discussion' (Jung et al, 2002: 153). Therefore as in 
traditional learning, motivation is not only based on individual factors. 

For the e-Learning solution and process, the key factors are 'interaction' and 'practice'. The importance of 
these factors was determined as a result of the coding of the factors that influence effectiveness as well as the 
codes including the reasons that e-Learning was or was not effective. Interrelated factors, such as 
'instructional scaffolding', 'modelling' and 'support', were combined into a single factor, 'interaction'. Though 
e-Learning is often considered to be equally or more effective than face-to-face learning, interaction is 
generally considered to be critical to the effectiveness of e-Learning, as illustrated in the following papers: 
'Students valued interaction with instructor as an important factor in online learning. New students had a 
better success rate in instructor-led online courses than in independent-study online courses. Adult students 
need modelling and scaffolding to be successful in an online environment' (Jiang, Parent and Eastmond, 2006); 
'...the supported training group had a significantly higher program completion rate than the independent 
group' (Bennett-Levy et al, 2012); and 'Results indicate teachers attributed improved student learning to 
technology use; online communication with peers and experts reduced teacher isolation, enhanced 
professional practice and gave access to perspectives and experiences otherwise unavailable; but the 
additional workload discouraged several teachers' (Hawkes and Good 2000). The last article also touches on 
the contextual factors as it emphasises the need to consider both the time available to the learners and the 
expected workload of the e-Learning (see also Noesgaard, 2014). 

A recurring factor contributing to e-Learning effectiveness is 'practice'. The learner is given the opportunity or 
is required to practice the educational material presented via the e-Learning solution in case studies, 
simulations or actual work situations. The aim is to support learning retention and transfer to practice, as 
simulated practice provides a safe learning environment before the skills are applied in critical work situations: 
'Intermediate-fidelity simulation is a useful training technique. It enables small groups of students to practice 
in a safe and controlled environment how to react adequately in a critical patient care situation. This type of 
training is very valuable to equip students with a minimum of technical and non-technical skills before they use 
them in practice settings' (Alinier et al, 2006). 
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Figure 2: Key factors influencing effectiveness 

The model in Figure 2 shows the key factors (in grey) that were identified by the review to influence the 
effectiveness of e-Learning (artefact) across the multiple definitions (Table 2). A supportive and resourceful 
learning environment (context) must be in place. The level of motivation of the individual(s) interacting with e- 
Learning (artefact) influences the time they spend using the product, and previous online or professional 
experience appears to have a largely positive impact on effectiveness. An e-Learning design that 
accommodates interaction between instructor(s) and peers and provides opportunities to practice the e- 
Learning material in simulated—or, when appropriate, real-life-work situations also promotes e-Learning 
effectiveness. As such, this model shows the interrelation of the key factors influencing effectiveness, but it 
cannot stand alone, as this abstraction like many papers in the literature review does not take the great variety 
of definitions into account. However, it is a good tool for starting discussions about effectiveness. 

4.1 Why is this important? 

Understanding the factors that are deemed critical for effective e-Learning in the context of adult learning can 
enable learning professionals to reflect on their priorities regarding learning design. The model and list of 
factors can be used as a checklist and a starting point for discussing how to ensure that the factors relevant to 
specific e-Learning participants, contexts and solutions are taken into account. 

5. Critical reflections 

Many of the studies analysed in this integrative review value e-Learning based on how well a given e-Learning 
solution accommodates for individual characteristics (experience and motivation) or the extent to which the 
solution provides opportunities for interaction and practice and to a smaller extent whether or not the 
necessary contextual resources and support are in place (Figure 2). 

As discussed, approximately half the papers included in the literature review use a comparative methodology. 
This means that the effectiveness of e-Learning is largely defined based on how well the e-Learning performed, 
compared to traditional face-to-face teaching with the same content. Thus, the same definitions of 
effectiveness are used for both e-Learning and face-to-face teaching, and e-Learning must outperform face-to- 
face teaching in order to be considered effective. With this in mind, it becomes important to take another 
critical look at the key factors, since the comparative research methodology affects their perceived 
importance. This means that some of the factors like interaction and practice may have surfaced as critical 
during the review, because these factors illustrate the strong sides of face-to-face teaching. A given e-Learning 
solution must, therefore, entail these factors to be effective when compared to face-to-face teaching. But do 
we want e-Learning to be more like face-to-face teaching? Is the most effective e-Learning an online 
replication of the classroom setting? What would happen if policy makers and researcher stopped asking if e- 
Learning is as good as what we currently have with face-to-face teaching and instead started asking if we could 
get something better or different from e-Learning? Should different modalities have the same measures of 
performance, or should we consider e-Learning to be a unique learning process and thus use different 
definitions of effectiveness? 
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Within recent years, massive open online courses (MOOCs) have received profound attention within the field 
of e-Learning. MOOCs offer free courses, competence development and even certification. They are often 
considered to be the promised land of education, democratising education through scalable technology. As a 
stand-alone solution, MOOCs provide opportunities for reflecting on and constructing new knowledge, but 
often they entail a minimum amount of live interaction. Many MOOCs continue to be online replications of 
classrooms primarily consisting of video lectures, multiple-choice quizzes, Q&As and more informal after-class 
discussions in online discussion forums. 

So are MOOCs ineffective because little interaction is provided compared to face-to-face teaching? And if not, 
is interaction not a key factor in e-Learning effectiveness? We argue that interaction is indeed essential to 
learning retention and learning transfer. An educational design applying collaboration and interaction with 
peers and a facilitator (the teacher or instructor) can provide a purposeful space for reflecting on the practice 
as well as an empathetic customisation of the subject matter, which no automatic process is capable of yet. 
Hence, when the overarching objective is to design e-Learning which increases learning retention and work 
performance, the effectiveness of e-Learning may very well be evaluated by the quality of the interaction 
provided. But what if the overarching objective is different than the definitions provided here (Table 2)? What 
if we, despite the vast number of definitions in this paper, are missing more e-Learning specific definitions that 
are independent of face-to-face teaching, such as democratization and scalability? And as a last mental 
exercise, what if we started evaluating face-to-face instructions on these objectives? 

6. Conclusions 

This paper has discussed the following research questions: How is the effectiveness of e-Learning defined? 
How is the effectiveness of e-Learning measured? What makes e-Learning solutions effective? 

Through an integrative review, relying on both a literature review and an empirical study, this paper identified 
multiple ways to define effectiveness, with 'learning outcome' as the most prominent definition. The authors 
highlighted the benefits of reflection on and clarification of the way in which these definitions are used in 
research and practice. The paper discovered that the research is largely applying quantitative and comparative 
methodologies. In this regard, this paper suggests that applying purely quantitative measures to fulfil 
predefined learning objectives does not allow researchers and practitioners to discover unexpected and 
unintended transfers to practice and presents potential sources of error. Including open-ended qualitative 
questions in surveys can substantially improve the validity of such approaches; the empirical study and some 
of the analysed papers show that self-assessment can give researchers and designers quality feedback on the 
effectiveness of the e-Learning solution. To understand what makes e-Learning solutions effective, this paper 
analysed factors promoting the effectiveness of e-Learning. These factors were categorised according to the 
context in which the e-Learning solution was used, the artefact (the e-Learning solution itself) and the 
individuals that used the artefact. Subsequently, further categorization into key factors resulted in a model to 
guide e-Learning design. 

One of the many questions raised in the discussion was the question of whether e-Learning and traditional 
face-to-face learning should be measured based on the same definitions of and approaches to effectiveness. 
To find appropriate answers to this question, the authors suggest that future researchers and designers 
critically consider the identified definitions, measures and factors when designing for effective e-Learning. 

7. Limitations and further work 

The literature study has certain limitations: First, the fact that the analysis is based primarily on abstracts, only 
including a few full papers, might have resulted in slightly different categorisations than had the full papers 
been analysed. Second, e-Learning is dealt with as a single concept, when in fact it consists of a great variety of 
delivery methods, technologies and learning designs. However, these variations are coded in the literature 
study, and the authors look forward to further exploring the concept of e-Learning. Third, the literature study 
considers a larger number of papers than most other literature studies. Still, the quantitative analysis will 
benefit from an increased number of coded abstracts since some of the findings are based on relatively small 
numbers (Table 2). However, as mentioned in the methodological section, the authors chose theoretical 
saturation over volume. Finally, both researchers were involved in discussions about the design of the e- 
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Learning solution tested in the empirical study and thus may be affected by effectiveness bias. The researchers 

will continue to explore these concerns as the study continues. 

8. Recommendations 

Based on the analyses in the previous chapters, the following recommendations are proposed: 

• There are many understandings of e-Learning effectiveness. Be sure to clarify what would make your 
solutions effective. Consider using the list of definitions for inspiration (Table 2). 

• Do not measure effectiveness simply for the sake of measuring. Know what measurements will give 
you the documentation your stakeholders require and the answers you need to continuously improve 
your solutions. 

• When designing e-Learning, consider the key factors that impact e-learning effectiveness (Figure 2). 

• Be critical and consider whether or not your face-to-face and e-Learning solutions should use 
different definitions of effectiveness (performance measures). Aim to compare your solution to other 
solutions using the same definitions of effectiveness. 
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